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EXECUTIVE  SUMMARY 


This  report  outlines  a  proposed  two  year  program  for  the  investigation  and  implementation  of  a  low 
cost  Automated  Operational  Flight  Performance  Monitoring  System  (AOFPMS)  for  CC130 
operations.  This  work  follows  on  from  Phase  I  of  the  study,  that  developed  a  performance 
scoring  system  for  CC130  flight  operations.  The  system  will  utilize  automated  data  collection  and 
transmission  technologies,  a  knowledge-based,  flight-performance  scoring  system,  and  advanced 
artificial  intelligence  and  data  fusion  technologies  for  the  automated  tracking  of  long-term  trends  in 
aircrew  performance  and  operational  flight  safety  during  changes  in  training,  doctrine,  operations, 
and  experience  levels. 
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INTRODUCTION 


This  Volume  is  the  third  in  a  six  volume  series  describing  a  human  factors  study  of  Canadian 
Forces  (CF)  CC-130  aircraft  operation  (see  References  [1-5]  for  details  of  the  remaining 
documents  in  this  series).  This  Volume  deals  with  the  issue  of  crew  performance  measurement. 

In  a  recent  AGARD  publication  [6],  it  was  projected  that  both  aircrew  flight  performance  and 
aircrew  psycho-physiological  state  would  be  routinely  monitored  by  the  year  2020,  both  to  track 
the  efficacy  of  the  crew  and  to  provide  input  into  the  artificial  intelligence  of  the  ‘electronic  crew¬ 
member’.  At  the  present  time  such  systems  have  been  tested  both  in  simulators  and  on  commercial 
transport  aircraft. 

Aircrew  performance  monitoring  or  measurement  can  be  defined  as  “...the  act  of  ascertaining  the 
dimensions,  quantity,  quality,  of  the  performance  of  the  aircrews  by  comparison  against  a 
standard...”  [7].  Performance  monitoring  can  be  done  during  aircrew  section,  aircrew  training, 
recurring  qualification,  and  role  qualification  [7],  as  well  as  during  ongoing  day-to-day  operations. 
The  current  extent  of  operational  flight  performance  monitoring  in  most  military  transport 
operations,  including  the  CF  CC-130  fleet,  consists  of  satisfactory/unsatisfactory  ratings  by 
‘check’  pilots.  With  respect  to  evaluations  by  check  pilots,  errors  are  largely  catalogued  with 
respect  to  procedural  or  technical  mistakes,  with  the  quantity  of  errors  as  the  primary  concern  and 
with  little  focus  on  the  magnitude  of  the  error  or  its  impact  on  the  safety  of  the  flight  [8] .  Statistics 
on  major  incidents  generally  come  from  pilot-based  narrative  reporting  systems  similar  to  the  US 
Aviation  Safety  Reporting  System  used  by  the  commercial  airline  sector.  Due  to  its  rarity,  accident 
data  is  of  limited  value  for  tracking  trends  in  composite  performance,  nor  does  it  quantify  the 
number  of  significant  and/or  dangerous  deviations  which  did  not  result  in  an  accident.  None  of 
these  data  sources  provide  sufficient  information  on  performance  to  correlate  with  changes  in 
procedures,  training,  or  experience,  nor  does  the  data  collected  quantify  the  extent  of  performance 
degradation  which  increases  the  risk  of  accidents. 

In  many  cases,  the  subjective  information  from  check  pilots  or  self-reports  by  aircrew  is  of  less 
value  than  objectively  measured  data,  such  as  that  obtained  from  flight-data  recorders  [8].  As  an 
example,  flight  crews  involved  in  controUed-ffight-into-terrain  (CFIT)  accidents  generally  know 
where  they  are  laterally,  but  not  vertically  [9],  but  almost  no  reliable  data  are  available  as  to  the 
frequency  at  which  flight  crews  deviate  from  the  correct  vertical  or  lateral  position.  Expert 
observer  evaluations  undertaken  during  research  programs  have  provided  valuable  insight  into 
crew  behaviour,  cockpit  resource  management  (or  lack  of),  and  how  flight  crews  resolve  the 
consequences  of  their  own  errors.  However,  for  routine  monitoring  and  the  quantification  of 
incidence  rates,  cost  would  prevent  such  an  approach. 

A  number  of  barriers  to  the  development  of  crew  performance  monitoring  systems  based  on  data 
collected  from  flight  data  recorders  remain.  The  tools  for  objectively  measuring  and  quantifying 
flight  performance  are  not  yet  mature,  cost  is  a  major  issue,  as  is  pilot  acceptance.  However,  there 
are  a  number  of  programs  in  use  by  commercial  operators  in  Europe,  including  British  Airways 
(BA),  Scandinavian  Airline  Systems  (SAS),  Royal  Dutch  Airlines  (KLM),  and  Air  Portugal  (TAP) 
which  track  flight  performance  on  a  continuous  basis,  providing  feedback  to  specific  crews  in 
some  circumstances.  However,  they  are  primarily  concerned  with  identifying  unwanted  events, 
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i.e.,  exceedences  or  incidents.  Little  attempt  is  made  to  track  overall  trends  in  group  performance, 
and  the  impact  of  training  and  procedural  changes  [10]. 

The  National  Aeronautics  and  Space  Administration  (NASA)  and  the  United  States  Federal 
Aviation  Administration  (FAA)  have  undertaken  the  multi-year  joint  development  of  an  Automated 
Performance  Measurement  System  (APMS)  to  generate  a  set  of  tools  and  methodologies  for  the 
commercial  airline  sector  in  the  United  States  to  collect  and  process  the  large  quantities  of  flight 
data  which  would  be  generated  on  an  ongoing  basis  [10].  The  conceptual  design  of  the  APMS 
was  used  as  a  basis  for  tailoring  an  approach  to  the  development  of  an  Automated  Operational 
Flight  Performance  Monitoring  System  (AOFPMS)  for  CC-130  operations. 

The  NASA/FAA  system  is  designed  to  analyse  the  extent  of  deviations  from  nominal  flight 
parameters  in  a  commercial  airline  industry.  As  part  of  the  desire  to  develop  a  flight-performance 
scoring  system  in  Phase  I  of  this  study  [11,  12],  that  was  both  relevant  to  CC-130  operations  and 
focused  on  the  safety  of  operations,  a  flight-performance  scoring  algorithm  was  developed  using  a 
knowledge  engineering  approach.  Twenty-nine  standards  officers  in  the  CF  Air  Transport  Group 
(ATG)  completed  a  survey  in  which  they  were  asked  to: 

(i)  rank  the  importance  of  each  flight  parameter  (i.e.,  altitude,  heading,  glide-slope  angle,  etc.)  in 
each  phase  of  a  flight, 

(ii)  assign  a  score  of  0%  to  the  absolute  deviation  from  each  nominal  flight  parameter,  which  in 
their  opinion,  would  result  in  a  significant  probability  of  an  accident,  and 

(iii)  generate  a  function  which  could  calculate  a  score  ranging  from  100%  to  0%  for  a  given 
deviation  from  the  optimal  flight  parameter. 

Composite  scoring  functions,  based  on  all  the  survey  results,  were  used  to  rank  the  performance  of 
15  crews  flying  a  simulated  ‘medivac’  mission.  The  performance  rankings  of  the  crews  were 
compared  to  the  proficiency  rankings  assigned  by  a  civilian  pilot  and  experienced  human  factors 
engineer  who  reviewed  the  cockpit  videos  of  the  simulated  flights.  The  flight  path  analysis  based 
on  the  scoring  functions  provided  a  ranking  of  the  crews  on  the  most  important  flight  parameters  in 
each  phase  of  the  flight  Scores  ranged  from  98%  to  51%  in  maintaining  assigned  altitude,  from 
66%  to  26%  in  maintaining  heading,  and  from  32%  to  15%  in  maintaining  glide  slope.  There  was 
a  significant  correlation  between  the  rankings  generated  from  the  glide  slope  scores  and  the 
proficiency  rankings  generated  by  the  check  pilot  (r  =  0.55,  p  <  0.04).  Crew  rankings  derived 
from  altitude  and  heading  scores  correlated  with  the  proficiency  ratings  atp  <  0.1. 

It  was  apparent  that  application  of  knowledge  engineered  scoring  system  (KESS)  to  an 
experimental  study,  with  a  small  number  of  crews,  flying  identical  missions,  in  a  simulator  setting, 
is  not  the  appropriate  use  of  such  a  technology.  Such  a  technique  would  more  appropriately 
address  the  needs  of  the  CC-130  community  where  it  can  be  used  to  track  the  kind  and  quantity  of 
errors  which  aircrew  make  and  provide  better  feedback  with  respect  to  training,  doctrine,  mission 
scheduling,  and  duty-rest  cycles  on  the  overall  group  performance. 
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AUTOMATED  OPERATIONAL  FLIGHT  PERFORMANCE 

MONITORING  SYSTEM 


An  Automated  Operational  Flight  Performance  Monitoring  System  is  one  component  of  the  Flight 
Operations  Quality  Assurance  Program  (FOQA)  under  consideration  for  commercial  operations  in 
the  United  States.  The  goal  of  an  AOFPMS,  in  the  context  of  CF  CC-130  operations,  is  to  provide 
the  CF  with  a  tool  to  track  the  safety  and  efficacy  of  flight  operations. 

Data  collected  from  AOFPMS  could  be  used  at  a  number  of  levels;  such  as 

•  evaluating  performance  of  individual  crews  on  specific  flights,  i.e.,  providing  an  automated 
check  ride  capability; 

•  composite  data  could  be  be  collected  and  analysed  during  various  training  and  operational 
exercises,  or  used  to  identify  particular  operations,  procedures,  aerodromes,  or  flight 
conditions  where  safety  is  highly  compromised;  or 

•  the  system  could  be  used  to  track  long  term  trends  in  flight  performance  and  safety  on  a 
squadron  or  fleet-wide  level  (considered  to  be  the  most  important,  and  statistically  valid,  use  of 
the  system). 

The  results  of  the  Phase  I  study  indicated  that  a  flight  performance  monitoring  system  based  on 
performance  criteria  provided  by  standards  and  training  pilots  from  the  CC-130  community  is 
possible.  This  Volume  outlines  an  approach  to  the  development  of  an  AOFPMS  which  will: 

•  allow  the  quantification  of  flight  performance  with  respect  to  flight  safety;  and 

•  track  changes  and  trends  in  flying  performance  as  a  result  of  changes  in  doctrine,  strategy, 
tactics,  training,  and  experience  levels. 

It  is  necessary  that  such  a  system  be  designed  and  implemented  specific  to  the  requirements  of  CC- 
130  operations  and  within  the  current  financial  restraints  imposed  on  the  Canadian  Forces.  A 
number  of  design  goals  have  been  used  to  devise  an  approach  to  the  development  of  an  AOFPMS. 
A  primary  focus  has  been  to  use: 

(i)  existing  hardware  and  software  development  tools  currently  available  at  DCIEM,  and 

(ii)  utilise  inexpensive  commercial  hardware  and  software  tools  to  design  and  implement  the 
system. 

Specific  design  goals  are  listed  in  the  following  section. 


Design  Goals 

•  The  system  should  provide  an  objective  measure  of  aircrew  flight  performance. 

•  Flight  performance  measures  should  be  based  on  the  expertise  and  experience  of 
standards  and  training  officers. 
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•  The  scoring  and  analysis  of  flight  performance  measures  should  focus  on  the  safety  of 
flight  operations. 

•  The  design  should  minimise  the  personnel  and  financial  resources  of  the  operators 
required  to  implement  and  run  the  system. 

•  The  system  should  be  applicable  to  the  complex  CC-130  operational  requirements  and 
missions. 

•  The  system  should  be  readily  affordable  in  terms  of  the  initial  capital  costs  and  long-term 
operating  costs. 

•  The  data  collection,  data  analysis,  and  report  generation  should  be  automated. 

•  A  high  degree  of  data  security  should  be  inherent  in  the  system  in  order  to  insure  the 
confidentiality  of  the  data. 


Developmental  Approach 

An  extensive  literature  review  and  trade  study  has  been  undertaken  into  the  hardware  and  software 
technologies  that  would  allow  these  design  goals  to  be  met.  A  two-year  development  effort  is 
proposed  building  on  the  technical  expertise  developed  in  Phase  I  of  the  CC-130  study  as  well  as 
DCIEM  resources  developed  in  support  of  other  research  and  development  programs. 

Year  I  efforts  will  focus  on  the  following  areas: 

Knowledge  Engineered  Scoring  System  (KESS).  In  Phase  I  of  this  study,  the 
Knowledge  Engineering  Scoring  System  was  developed  using  a  written  questionnaire  which  was 
distributed  to  29  standards  and  training  pilots  in  the  ATG  community.  Based  on  the  feedback 
received,  a  revised  computer-based  questionnaire  would  be  prepared  and  distributed.  Follow-up 
interviews  would  also  be  undertaken  in  order  to  develop  a  consensus  as  to  the  appropriate  level  of 
risk  that  would  be  assigned  to  a  given  flight  deviation.  This  modified  KESS  would  be 
incorporated  into  the  AOFPMS. 

Statistical  Analysis/Artificial  Intelligence  Technologies.  Techniques  for  tracking  and 
analysing  trends  in  various  performance  metrics  will  be  further  investigated.  Prototype  versions  of 
the  software  will  be  implemented  and  tested  using  flight  data  collected  from  commercial  aircraft 
operations.  Techniques  for  developing  a  composite  performance  score  based  on  multiple  attributes 
of  flight  performance  [13]  will  be  implemented  and  tested  using  the  data  from  Phase  I  of  the  study 
as  well  as  flight  data  records  from  commercial  operations  and  published  research. 

In  the  initial  implementation  of  the  system,  performance  tracking  will  focus  on  the  take-off  and 
landing  phases  or  other  phases  of  flight,  such  as  search  patterns,  which  provide  appropriate 
benchmark  performance  with  which  to  compare  actual  flight  profiles.  These  segments  of  flight  are 
also  the  most  critical  with  respect  to  flight  safety.  A  major  part  of  this  study  will  be  to  determine 
which  flight  data  can  be  automatically  compared  to  an  objective  standard. 

Database  and  Data  Warehouse.  Implementation  of  an  AOFPMS  will  result  in  the  collection 
of  tera-byte  quantities  of  data.  Costs  for  the  physical  storage  of  such  quantities  have  been 
dramatically  reduced  with  advances  in  optical  storage  technologies.  However,  the  organisation  and 
tracking  of  such  large  quantities  of  data  is  problematic.  Commercial  database  systems,  designed 
primarily  for  financial  and  personnel  data  are  generally  inappropriate  for  the  type  of  data  that  would 
be  collected  from  an  aircraft  flight  data  recorders.  DCIEM  has  investigated  a  number  of  database 
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approaches  developed  in  support  of  remote  sensing  systems  and  interplanetary  space  missions. 
This  has  resulted  in  several  prototype  database  systems  for  the  storage  and  archiving  of  large 
amounts  of  physiological  research  data.  A  prototype  database  and  data  warehouse  system  utilising 
object-orientated  database  management  technologies  will  be  designed  to  handle  the  large  amount  of 
data  from  CC-130  operations.  As  in  the  case  of  the  physiological  data,  the  data  storage  system  will 
be  integrated  into  the  data  analysis  architecture  of  the  AOFPMS  for  efficient  analysis  and  report 
generation. 


Flight  data  recorder 

A  major  barrier  to  implementing  an  AOFPMS  in  the  CC-130  operational  system  may  be  the  cost  of 
retrofitting  and  maintaining  a  dedicated  AOFPMS  flight  data  recording  system  in  the  CC-130  fleet. 
Additional  costs  associated  with  transferring  the  data  to  an  analysis  centre  could  also  be 
prohibitive.  However,  advances  in  data  collection  and  storage  technologies  have  reduced  potential 
costs  dramatically.  A  trade  study  and  preliminary  design  of  an  inexpensive  flight  data  collection 
module  to  interface  with  the  databus  of  the  upgraded  avionics  system  of  the  CC-130  aircraft  will  be 
undertaken.  The  system  will  be  based  on  commercial  embedded  digital  hardware  for  local  signal 
processing  and  data  storage  and  modem  cellular  telephony  hardware  and  software  to  automate  the 
transfer  of  flight  data  to  DC3EM  for  processing  and  analysis.  As  much  local  data  processing  and 
analysis  would  be  performed  in  the  aircraft-based  system  as  possible  to  reduce  the  quantity  of  data 
transferred  and  the  associated  costs. 

At  the  end  of  the  first  year,  technical  reports  will  be  prepared  on  the  feasibility  and  progress  in  each 
of  the  key  technical  areas,  described  above,  required  for  the  implementation  of  the  system.  Given 
sufficient  progress  in  addressing  all  of  the  implementation  problems,  the  second  year  of  the 
program  will  involve  the  development  and  testing  of  a  prototype  AOFPMS  system. 

Flight  data  recorder.  A  prototype  flight  data  collection  and  transmission  unit  would  be 
installed  in  one  of  the  CC-130  aircraft.  The  automated  data  collection,  security  systems,  and 
transfer  of  flight  data  to  DCIEM  would  be  tested. 

Statistical  Analysis/AI  Technologies.  Software  for  the  automated  processing  of  flight  data 
would  be  tested,  evaluated,  and  refined.  The  overall  efficacy  of  the  data  analysis  technologies  and 
report  generation  will  be  evaluated.  The  operators  will  be  asked  for  input  on  the  usefulness  of  the 
analysis  and  the  design  of  the  automated  reports. 

Data  Base  and  Data  Warehouse.  The  design  of  database  and  data  warehouse  system  will  be 
finalised.  A  prototype  version  of  the  software  will  be  implemented  and  tested  on  the  DCIEM  file 
server. 

At  the  end  of  the  second  year  of  the  study,  a  second  series  of  reports  will  be  prepared.  These  will 
include  recommendations  as  to  whether  to  proceed  with  implementation  of  the  AOFPMS  in  the 
CC-130  fleet  based  on  feedback  from  the  operators.  This  report  will  include  estimates  of  the  cost 
of  hardware  installation  in  the  CC-130  aircraft,  monetary  and  manpower  costs  associated  with  the 
ongoing  maintenance  of  the  hardware,  and  the  ongoing  costs  associated  with  maintaining  the 
analysis  systems  and  data  warehouse  at  DCIEM. 
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CONCLUSIONS 


An  Automated  Operational  Flight  Performance  Monitoring  System  (AOFPMS)  could  be  a  valuable 
tool  for  the  quantification  of  flight  performance  with  respect  to  flight  safety;  and  for  the  tracking  of 
changes  and  trends  in  flying  performance  as  a  result  of  modifications  to  doctrine,  strategy,  tactics, 
training,  and  experience  levels  of  CC130  personnel.  It  is  necessary  that  such  a  system  be  designed 
and  implemented  specific  to  the  requirements  of  CC-130  operations  and  within  the  current  financial 
restraints  imposed  on  the  Canadian  Forces. 

A  two-year  development  effort  is  proposed  building  on  the  technical  expertise  developed  in  Phase  I 
of  the  CC-130  study  as  well  as  DCXEM  resources  developed  in  support  of  other  research  and 
development  programs.  This  approach  is  designed  to  minimise  the  risks  and  costs  associated  with 
the  development  of  an  AOFPMS. 
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